A theory of operation is a description of how a device or system should work. It is often included in documentation, especially maintenance/service documentation, or a user manual. It aids troubleshooting by providing the troubleshooter with a mental model of how the system is supposed to work. The troubleshooter can then more easily identify discrepancies, to aid diagnosis of problem.
IBM Redbooks[1] are "Theories of operation" of their products.